{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Demo: Summarize and search through long reddit posts using dlt, Notion, and LanceDB\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zT23N_DetMD1"
   },
   "source": [
    "\n",
    "If you have the attention span to read those extra-long Reddit posts, you deserve respect. If you don't, then you deserve this demo.\n",
    "\n",
    "By the end of this 100% free demo, you'll have something like this, without needing to be a Python pro (well, not the happiest example... 👀):\n",
    "\n",
    "![image](https://storage.cloud.google.com/dlt-blog-images/demo_notebook_tuba.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dr-vHBTWykko"
   },
   "source": [
    "### **So what exactly is this Colab for?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xBKt-4U1zsp_"
   },
   "source": [
    "\n",
    "**TL;DR:** You'll learn how to automatically load AI summarized content from a specific subreddit into Notion, making content management and review more efficient for creators.\n",
    "\n",
    "![Overview](https://storage.googleapis.com/dlt-blog-images/notebook_tuba_demo_overview.png)\n",
    "\n",
    "\n",
    "**The full scoop:**\n",
    "\n",
    "- This notebook is your testament to the fact that YES, you can indeed automate the summary of those never-ending Reddit posts and park them neatly into Notion, all without spending a dime.\n",
    "- Consider this a `one-stop-shop template to breeze through content from any subreddit` — because, let’s face it, nobody has the time to read that much anymore.\n",
    "- If you fancy a bit of coding, customize your data source and tweak this setup to do anything else AI might handle — like:\n",
    "    - Bulk loading comments for sentiment analysis.\n",
    "    - Automating translations across any language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jPR3BsBDy1MQ"
   },
   "source": [
    "### **The coding corner**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UThijSdozFXU"
   },
   "source": [
    "**1. Install and import necessary libraries**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5YSvYrWrtNuo",
    "outputId": "895773f2-1357-4aa0-8f46-8b1fe63bfcc1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting praw\n",
      "  Downloading praw-7.7.1-py3-none-any.whl (191 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m191.0/191.0 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting notion_client\n",
      "  Downloading notion_client-2.2.1-py2.py3-none-any.whl (13 kB)\n",
      "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (3.8.1)\n",
      "Requirement already satisfied: dlt in /usr/local/lib/python3.10/dist-packages (0.5.1)\n",
      "Collecting prawcore<3,>=2.1 (from praw)\n",
      "  Downloading prawcore-2.4.0-py3-none-any.whl (17 kB)\n",
      "Collecting update-checker>=0.18 (from praw)\n",
      "  Downloading update_checker-0.18.0-py3-none-any.whl (7.0 kB)\n",
      "Requirement already satisfied: websocket-client>=0.54.0 in /usr/local/lib/python3.10/dist-packages (from praw) (1.8.0)\n",
      "Collecting httpx>=0.15.0 (from notion_client)\n",
      "  Downloading httpx-0.27.0-py3-none-any.whl (75 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk) (8.1.7)\n",
      "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk) (1.4.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk) (2024.5.15)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk) (4.66.4)\n",
      "Requirement already satisfied: PyYAML>=5.4.1 in /usr/local/lib/python3.10/dist-packages (from dlt) (6.0.1)\n",
      "Requirement already satisfied: astunparse>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from dlt) (1.6.3)\n",
      "Requirement already satisfied: fsspec>=2022.4.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (2023.6.0)\n",
      "Requirement already satisfied: gitpython>=3.1.29 in /usr/local/lib/python3.10/dist-packages (from dlt) (3.1.43)\n",
      "Requirement already satisfied: giturlparse>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (0.12.0)\n",
      "Requirement already satisfied: hexbytes>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from dlt) (1.2.1)\n",
      "Requirement already satisfied: humanize>=4.4.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (4.7.0)\n",
      "Requirement already satisfied: jsonpath-ng>=1.5.3 in /usr/local/lib/python3.10/dist-packages (from dlt) (1.6.1)\n",
      "Requirement already satisfied: makefun>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (1.15.4)\n",
      "Requirement already satisfied: orjson!=3.10.1,!=3.9.11,!=3.9.12,!=3.9.13,!=3.9.14,<4,>=3.6.7 in /usr/local/lib/python3.10/dist-packages (from dlt) (3.10.6)\n",
      "Requirement already satisfied: packaging>=21.1 in /usr/local/lib/python3.10/dist-packages (from dlt) (24.1)\n",
      "Requirement already satisfied: pathvalidate>=2.5.2 in /usr/local/lib/python3.10/dist-packages (from dlt) (3.2.0)\n",
      "Requirement already satisfied: pendulum>=2.1.2 in /usr/local/lib/python3.10/dist-packages (from dlt) (3.0.0)\n",
      "Requirement already satisfied: pytz>=2022.6 in /usr/local/lib/python3.10/dist-packages (from dlt) (2023.4)\n",
      "Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (2.31.0)\n",
      "Requirement already satisfied: requirements-parser>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (0.9.0)\n",
      "Requirement already satisfied: semver>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (3.0.2)\n",
      "Requirement already satisfied: setuptools>=65.6.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (67.7.2)\n",
      "Requirement already satisfied: simplejson>=3.17.5 in /usr/local/lib/python3.10/dist-packages (from dlt) (3.19.2)\n",
      "Requirement already satisfied: tenacity>=8.0.2 in /usr/local/lib/python3.10/dist-packages (from dlt) (8.5.0)\n",
      "Requirement already satisfied: tomlkit>=0.11.3 in /usr/local/lib/python3.10/dist-packages (from dlt) (0.13.0)\n",
      "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from dlt) (4.12.2)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from dlt) (2024.1)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.3->dlt) (0.43.0)\n",
      "Requirement already satisfied: six<2.0,>=1.6.1 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.3->dlt) (1.16.0)\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.10/dist-packages (from gitpython>=3.1.29->dlt) (4.0.11)\n",
      "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.15.0->notion_client) (3.7.1)\n",
      "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx>=0.15.0->notion_client) (2024.7.4)\n",
      "Collecting httpcore==1.* (from httpx>=0.15.0->notion_client)\n",
      "  Downloading httpcore-1.0.5-py3-none-any.whl (77 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx>=0.15.0->notion_client) (3.7)\n",
      "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.15.0->notion_client) (1.3.1)\n",
      "Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx>=0.15.0->notion_client)\n",
      "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: ply in /usr/local/lib/python3.10/dist-packages (from jsonpath-ng>=1.5.3->dlt) (3.11)\n",
      "Requirement already satisfied: python-dateutil>=2.6 in /usr/local/lib/python3.10/dist-packages (from pendulum>=2.1.2->dlt) (2.8.2)\n",
      "Requirement already satisfied: time-machine>=2.6.0 in /usr/local/lib/python3.10/dist-packages (from pendulum>=2.1.2->dlt) (2.14.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->dlt) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->dlt) (2.0.7)\n",
      "Requirement already satisfied: types-setuptools>=69.1.0 in /usr/local/lib/python3.10/dist-packages (from requirements-parser>=0.5.0->dlt) (70.3.0.20240710)\n",
      "Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from gitdb<5,>=4.0.1->gitpython>=3.1.29->dlt) (5.0.1)\n",
      "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx>=0.15.0->notion_client) (1.2.2)\n",
      "Installing collected packages: h11, update-checker, prawcore, httpcore, praw, httpx, notion_client\n",
      "Successfully installed h11-0.14.0 httpcore-1.0.5 httpx-0.27.0 notion_client-2.2.1 praw-7.7.1 prawcore-2.4.0 update-checker-0.18.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    }
   ],
   "source": [
    "!pip install praw notion_client nltk dlt\n",
    "\n",
    "# Standard library imports\n",
    "import datetime\n",
    "import os\n",
    "\n",
    "# Related third party imports\n",
    "import praw\n",
    "import toml\n",
    "from notion_client import Client\n",
    "from transformers import pipeline\n",
    "import nltk\n",
    "\n",
    "nltk.download(\"punkt\")\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "# Data Load Tool\n",
    "import dlt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6nKAccAex1E-"
   },
   "source": [
    "**2. Initialize the PRAW (Python Reddit API Wrapper) client and the summarizer using Facebook's BART model:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 333,
     "referenced_widgets": [
      "09ccbb1123ec46a18f64ace4723d1ae5",
      "ec75896fc55d4490ae243077002022d2",
      "1f9a588f68c846b584bef8fa07b9cddf",
      "e572b4d7849c4844aed8f25bb2f96bdb",
      "5b42ef7ed762447faef272fe473d8c6f",
      "3ef8e558298446a899256aee7019d25b",
      "489aad35039548c1a0f4f4a48013b9c0",
      "be4926d663b4413685e19e4b0d796722",
      "2439f2f082314c1a8cda21d479bd3643",
      "c8da4dcf3de74e4b940b2901bc0840ea",
      "536a2cab6a284683a9fafacaab46d385",
      "25faf9d0408645d1becd39fcd2d1ce95",
      "11d2820c9676421eb6a1e4944aff95b4",
      "3dcdff020338480094c5e410d2f82a1f",
      "cb2794abb89c47f387a825717e435670",
      "19708415c39f4ea0aaea26bf27498b49",
      "77ca6047dafc45de9f86f3107102088d",
      "446adb40ce834e4e9137e3d444075cd6",
      "21b2b77ed18c4abb9ef33035b6e1f2f4",
      "2f76fb273a3b43a19346767937001595",
      "025432407a36450b90f9d661ac0c8d2e",
      "81fbf170a65c49118399a88369fecdb4",
      "7f7fabc4f21a43adac1c00c075dfe76d",
      "4d062614f9424a3b8125628544bc8d92",
      "22834af24a464a0dbaccc22eeeedfdf7",
      "f6967b1d432f430e80764f6acecf8500",
      "38169afbeddb48419efad7353a4b2a5c",
      "82ba8ab4a48546b8923c66fadcc0d7ed",
      "742fddf6bac44dd3b1b8c2dbf8f85757",
      "d39e3f842d624e6fa9f38f3fdd5d38c9",
      "3c83c6f621f249288421c4f207471d30",
      "002067540b9b4cdd84c4b6783ef66750",
      "b8f0f762c8ff4a8080f4f4457d2dbfc6",
      "648ceb9493f24f2ba06672025914a4f0",
      "d09b6fe7c57940fcacbee729f7f07d4b",
      "5ced7b584f7742f0b8207085d8dbd1d5",
      "bfe0633cbb2d45c8971ee28e694759d5",
      "883ffe757da544e89ee2a846f6b2e50b",
      "331768f93c7b411fa1c0d73f63f4aff3",
      "fffc9d8e4f7f484ea618628a252c99a3",
      "ef72d536ea72456c868753968d6ccd0d",
      "0b06b40d0d684357801688632856a6c1",
      "7f8416b61c8d4406b163c7e17f387931",
      "8dd6c26c23df472b8236d0172f6621e7",
      "1264b306d32a43fab2bc20046508da13",
      "c831782c9dfc472e9577e9e07b4b40e1",
      "f99ded668bf548a886fa95c9043a768c",
      "d44b89ac088e43718787a76a2c7e4e17",
      "d4afa02f3b02463c816dcc362385f2ab",
      "0405440b52c64181a4bcea9021ac84b3",
      "f31586171a4c4e34959929bad945263c",
      "f566b5bde5b44fc2a3cb2303a4d1bc01",
      "e6e37d07b8884a4e89c7c6dd4af80b7c",
      "2d72981657fa4aba9fdfa6adc5519c69",
      "78599b39629e4ef380cdf9b0c5bf014d",
      "2fa13b3a7c34415c9e0be2609a838156",
      "1ec55f31621c47b8bdf763351938400d",
      "8b9f13dfb0eb46eaae11f1a4f0a1196e",
      "f6998c78a6d242abb3f948e05e05e8f2",
      "854ca26c59174406b372a2478f39b9ac",
      "75ac1d053b184f9294a52269a9316039",
      "cde93f06f21a43bb94e94c8bb1160d36",
      "59475be56cf5416a8ebbbf2d720bd79f",
      "954cff1002ec4f508dec6d4896ab3c53",
      "7a4602c01d7c4b8dbc3afecffa344112",
      "c4426edabd704ec2a38e82c2c9e16fef"
     ]
    },
    "id": "uWi3oIQRt3nn",
    "outputId": "6753824e-c12d-4561-9291-8daf24606753"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09ccbb1123ec46a18f64ace4723d1ae5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/1.58k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "25faf9d0408645d1becd39fcd2d1ce95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/1.63G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f7fabc4f21a43adac1c00c075dfe76d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "generation_config.json:   0%|          | 0.00/363 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "648ceb9493f24f2ba06672025914a4f0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/899k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1264b306d32a43fab2bc20046508da13",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2fa13b3a7c34415c9e0be2609a838156",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from google.colab import userdata\n",
    "\n",
    "userdata.get(\"REDDIT_CLIENT_ID\")\n",
    "\n",
    "reddit = praw.Reddit(\n",
    "    client_id=userdata.get(\"REDDIT_CLIENT_ID\"),\n",
    "    client_secret=userdata.get(\"REDDIT_SECRET\"),\n",
    "    password=userdata.get(\"REDDIT_PASSWORD\"),\n",
    "    user_agent=userdata.get(\"REDDIT_USER_AGENT\"),\n",
    "    username=userdata.get(\"REDDIT_USERNAME\"),\n",
    ")\n",
    "\n",
    "summarizer = pipeline(\"summarization\", model=\"facebook/bart-large-cnn\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8RcBtbuvA4-u"
   },
   "source": [
    "**3. Define helper functions:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jTJoD26VzVOK"
   },
   "outputs": [],
   "source": [
    "# This is just a helper function to convert a UNIX timestamp to an ISO 8601 formatted string with a UTC timezone indicator\n",
    "def unix_to_iso8601(unix_timestamp):\n",
    "    utc_datetime = datetime.datetime.fromtimestamp(unix_timestamp)\n",
    "    return utc_datetime.isoformat() + \"Z\"\n",
    "\n",
    "\n",
    "# This is just a helper function to determine the maximum and minimum length requirements for the summaries\n",
    "def dynamic_summary_length(text, max_ratio=0.4, min_ratio=0.2, min_length=30):\n",
    "    text_length = len(text.split())\n",
    "    max_length = max(min_length, int(text_length * max_ratio))\n",
    "    min_length = max(min_length, int(text_length * min_ratio))\n",
    "    return max_length, min_length\n",
    "\n",
    "\n",
    "# This is just a helper function to summarize text using the BART model\n",
    "def summarize_text(text):\n",
    "    try:\n",
    "        # Try summarizing the entire text first\n",
    "        max_length, min_length = dynamic_summary_length(text)\n",
    "        summary_object = summarizer(\n",
    "            text, max_length=max_length, min_length=min_length, do_sample=False\n",
    "        )\n",
    "        return summary_object[0][\"summary_text\"]\n",
    "    except Exception as e:\n",
    "        print(f\"Summarization failed: {e}. Splitting the text.\")\n",
    "\n",
    "        # Split the text into sentences\n",
    "        sentences = sent_tokenize(text)\n",
    "\n",
    "        # Find the midpoint in terms of the number of sentences\n",
    "        mid_point = len(sentences) // 2\n",
    "\n",
    "        # Split the text into two halves at the midpoint\n",
    "        first_half = \" \".join(sentences[:mid_point])\n",
    "        second_half = \" \".join(sentences[mid_point:])\n",
    "\n",
    "        # Summarize each half separately\n",
    "        try:\n",
    "            first_max_length, first_min_length = dynamic_summary_length(first_half)\n",
    "            first_half_summary = summarizer(\n",
    "                first_half,\n",
    "                max_length=first_max_length,\n",
    "                min_length=first_min_length,\n",
    "                do_sample=False,\n",
    "            )[0][\"summary_text\"]\n",
    "        except Exception as sub_e:\n",
    "            print(f\"First half summarization failed: {sub_e}\")\n",
    "            first_half_summary = first_half\n",
    "\n",
    "        try:\n",
    "            second_max_length, second_min_length = dynamic_summary_length(second_half)\n",
    "            second_half_summary = summarizer(\n",
    "                second_half,\n",
    "                max_length=second_max_length,\n",
    "                min_length=second_min_length,\n",
    "                do_sample=False,\n",
    "            )[0][\"summary_text\"]\n",
    "        except Exception as sub_e:\n",
    "            print(f\"Second half summarization failed: {sub_e}\")\n",
    "            second_half_summary = second_half\n",
    "\n",
    "        # Combine the summaries of both halves\n",
    "        combined_summary = first_half_summary + \" \" + second_half_summary\n",
    "        return combined_summary.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wS0WVNFAzZZF"
   },
   "source": [
    "\n",
    "\n",
    "```\n",
    "# This is formatted as code\n",
    "```\n",
    "\n",
    "**4. Define your custom `dlt` resource:**\n",
    "\n",
    "In the function below, we are using `dlt.sources.incremental` to perform incremental loading. It is used to track a specific field in the data source, in this case, the `Created_utc` field, which represents the time when a post was created.\n",
    "\n",
    "The initial_value parameter is set to \"1970-01-01T00:00:00Z\", which is the start of the Unix epoch time. This means that on the first run of the pipeline, it will load all posts since this time.\n",
    "\n",
    "On subsequent runs, `dlt.sources.incremental` will keep track of the maximum Created_utc value that it has seen, and only load posts that have a Created_utc value greater than this. This is how it achieves incremental loading: by only loading new data that has been created since the last run.\n",
    "\n",
    "Without using this functionality of `dlt`, you would have to manually keep track of the last `Created_utc` value that you have seen, and manually filter the posts to only include those that are newer. This would involve more complex code and potentially error-prone manual tracking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WZJMinlwzaas"
   },
   "outputs": [],
   "source": [
    "# Define the `primary_key` and set the `write_disposition` to `merge` for incremental loading\n",
    "@dlt.resource(primary_key=\"ID\", write_disposition=\"merge\")\n",
    "def subreddit_posts(\n",
    "    subreddit_name,\n",
    "    updated_at=dlt.sources.incremental(\n",
    "        \"Created_utc\", initial_value=\"1970-01-01T00:00:00Z\"\n",
    "    ),\n",
    "):\n",
    "    # Access the specified subreddit\n",
    "    subreddit = reddit.subreddit(subreddit_name)\n",
    "\n",
    "    # Retrieve the top 50 posts from the subreddit, adjust limit as necessary\n",
    "    top_posts = subreddit.top(limit=10)\n",
    "\n",
    "    for post in top_posts:\n",
    "        # Check if the post has text, summarize it if it does\n",
    "        if post.selftext:\n",
    "            summary = summarize_text(post.selftext)\n",
    "            print(\"Summarization successful!\")\n",
    "        else:\n",
    "            summary = \"No Text\"  # Handle posts without text\n",
    "\n",
    "        # Convert the post's creation time from UNIX timestamp to ISO 8601 format\n",
    "        created_time = unix_to_iso8601(post.created_utc)\n",
    "\n",
    "        # Yield the post data in a structured format\n",
    "        yield {\n",
    "            \"Title\": post.title,\n",
    "            \"ID\": post.id,\n",
    "            \"URL\": post.url,\n",
    "            \"Summary\": summary,\n",
    "            \"Created_utc\": created_time,\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bep68ZyRzfRJ"
   },
   "source": [
    "**5. Define Notion as a custom `dlt` destination:**\n",
    "\n",
    "While `dlt` supports a variety of regularly tested integrations, Notion is typically used as a data source and does not have built-in support as a destination within `dlt`. For guidance on using Notion as a source, refer to the [official documentation](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion). However, considering the wide variety of custom destinations available, configuring Notion as a custom destination provides a learning opportunity to effectively utilize `dlt`.\n",
    "\n",
    "It's important to note that if you have configured a `dlt` resource with incremental loading, you must also define your destination as a `dlt` destination to ensure the incremental loading functions correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ifsriwk_zgQg"
   },
   "outputs": [],
   "source": [
    "# Define the destination function for creating entries in the Notion database\n",
    "@dlt.destination(name=\"Notion\")\n",
    "def notion_create_post(items, table) -> None:\n",
    "    # Initialize the Notion client with the authentication secret from secrets\n",
    "    notion_client = Client(auth=userdata.get(\"NOTION_AUTHENTICATION\"))\n",
    "    # Retrieve the database ID from secrets\n",
    "    notion_db_id = userdata.get(\"NOTION_DATABASE_ID\")\n",
    "\n",
    "    # Iterate over each item to create an entry in the Notion database\n",
    "    for item in items:\n",
    "        notion_client.pages.create(\n",
    "            parent={\"database_id\": notion_db_id},\n",
    "            properties={\n",
    "                \"Title\": {\"title\": [{\"text\": {\"content\": item[\"Title\"]}}]},\n",
    "                \"ID\": {\"rich_text\": [{\"text\": {\"content\": item[\"ID\"]}}]},\n",
    "                \"URL\": {\"url\": item[\"URL\"]},\n",
    "                \"Summary\": {\"rich_text\": [{\"text\": {\"content\": item[\"Summary\"]}}]},\n",
    "                \"Created_utc\": {\n",
    "                    \"rich_text\": [{\"text\": {\"content\": str(item[\"Created_utc\"])}}]\n",
    "                },\n",
    "            },\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oDoAppCOzjmp"
   },
   "source": [
    "**6. Create and run your `dlt` pipeline:**\n",
    "\n",
    "Upon executing the code snippet below, your Notion database will be populated with basic information and summaries of subreddit posts. Utilizing incremental loading ensures that subsequent executions do not create duplicate entries.\n",
    "\n",
    "To explore different content, simply change the `subreddit_name` argument in the `subreddit_posts` function your `dlt` pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "72qkXE22znA6",
    "outputId": "1b6f8a34-9d33-4d23-bbc6-4034e739723d"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:praw:It appears that you are using PRAW in an asynchronous environment.\n",
      "It is strongly recommended to use Async PRAW: https://asyncpraw.readthedocs.io.\n",
      "See https://praw.readthedocs.io/en/latest/getting_started/multiple_instances.html#discord-bots-and-asynchronous-environments for more info.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summarization successful!\n",
      "Summarization successful!\n",
      "Summarization successful!\n",
      "Summarization successful!\n",
      "Summarization successful!\n",
      "Summarization successful!\n",
      "Summarization successful!\n",
      "Summarization failed: index out of range in self. Splitting the text.\n",
      "Summarization successful!\n",
      "Summarization failed: index out of range in self. Splitting the text.\n",
      "Summarization successful!\n",
      "Summarization successful!\n",
      "Pipeline reddit_notion_pipeline load step completed in 2.26 seconds\n",
      "1 load package(s) were loaded to destination Notion and into dataset None\n",
      "The Notion destination used <dlt.common.configuration.specs.base_configuration.CredentialsConfiguration object at 0x7a56f1d35270> location to store data\n",
      "Load package 1721134597.1803734 is LOADED and contains no failed jobs\n"
     ]
    }
   ],
   "source": [
    "# Create your dlt pipeline\n",
    "notion_pipeline = dlt.pipeline(\n",
    "    pipeline_name=\"reddit_notion_pipeline\", destination=notion_create_post\n",
    ")\n",
    "\n",
    "# Run your dlt pipeline\n",
    "load_info = notion_pipeline.run(subreddit_posts(subreddit_name=\"offmychest\"))\n",
    "print(load_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-bwsXE0eKgTf"
   },
   "source": [
    "---\n",
    "# **Good Things Come to Those who Finish Code Demos...**\n",
    "---\n",
    "\n",
    "Congrats on having a reasonably long attention span! 😆\n",
    "\n",
    "In this part, you'll do some additional cool stuff with the same Reddit data using [LanceDB](https://lancedb.github.io/lancedb/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DCiDIq_bziqd"
   },
   "source": [
    "### **What cool stuff?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KIZyMf310lIg"
   },
   "source": [
    "**TL;DR:** You'll basically have your own mini search engine for querying Subreddit post summaries.\n",
    "\n",
    "![Overview](https://storage.googleapis.com/dlt-blog-images/notebook_tuba_demo_overview_lancedb_corrected.png)\n",
    "\n",
    "**The full scoop:**\n",
    "\n",
    "- If you've never had the chance to work with vector databases, this is your calling.\n",
    "- Otherwise, this is a template to streamline your vector data pipelines with `dlt` and `LanceDB` - both open-source!\n",
    "- If you're up for more advanced Machine Learning tasks, this is a great starting point where you don’t need to worry about the data loading part."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qkGvALNVAAdW"
   },
   "source": [
    "## **The coding corner**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QOwx4g0GAN7L"
   },
   "source": [
    "**1. Install and import necessary libraries**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "z6cTIzKVzWYk",
    "outputId": "379a4cec-8ebe-46fc-d8ae-283d7a398506"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting dlt[lancedb]\n",
      "  Downloading dlt-0.5.1-py3-none-any.whl (712 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m712.3/712.3 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting lancedb\n",
      "  Downloading lancedb-0.10.1-cp38-abi3-manylinux_2_28_x86_64.whl (21.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m16.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: PyYAML>=5.4.1 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (6.0.1)\n",
      "Requirement already satisfied: astunparse>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (1.6.3)\n",
      "Requirement already satisfied: click>=7.1 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (8.1.7)\n",
      "Requirement already satisfied: fsspec>=2022.4.0 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (2023.6.0)\n",
      "Collecting gitpython>=3.1.29 (from dlt[lancedb])\n",
      "  Downloading GitPython-3.1.43-py3-none-any.whl (207 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.3/207.3 kB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting giturlparse>=0.10.0 (from dlt[lancedb])\n",
      "  Downloading giturlparse-0.12.0-py2.py3-none-any.whl (15 kB)\n",
      "Collecting hexbytes>=0.2.2 (from dlt[lancedb])\n",
      "  Downloading hexbytes-1.2.1-py3-none-any.whl (5.2 kB)\n",
      "Requirement already satisfied: humanize>=4.4.0 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (4.7.0)\n",
      "Collecting jsonpath-ng>=1.5.3 (from dlt[lancedb])\n",
      "  Downloading jsonpath_ng-1.6.1-py3-none-any.whl (29 kB)\n",
      "Collecting makefun>=1.15.0 (from dlt[lancedb])\n",
      "  Downloading makefun-1.15.4-py2.py3-none-any.whl (23 kB)\n",
      "Collecting orjson!=3.10.1,!=3.9.11,!=3.9.12,!=3.9.13,!=3.9.14,<4,>=3.6.7 (from dlt[lancedb])\n",
      "  Downloading orjson-3.10.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m141.1/141.1 kB\u001b[0m \u001b[31m981.0 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: packaging>=21.1 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (24.1)\n",
      "Collecting pathvalidate>=2.5.2 (from dlt[lancedb])\n",
      "  Downloading pathvalidate-3.2.0-py3-none-any.whl (23 kB)\n",
      "Collecting pendulum>=2.1.2 (from dlt[lancedb])\n",
      "  Downloading pendulum-3.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (384 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m384.9/384.9 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: pytz>=2022.6 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (2023.4)\n",
      "Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (2.31.0)\n",
      "Requirement already satisfied: requirements-parser>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (0.9.0)\n",
      "Collecting semver>=2.13.0 (from dlt[lancedb])\n",
      "  Downloading semver-3.0.2-py3-none-any.whl (17 kB)\n",
      "Requirement already satisfied: setuptools>=65.6.0 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (67.7.2)\n",
      "Collecting simplejson>=3.17.5 (from dlt[lancedb])\n",
      "  Downloading simplejson-3.19.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (137 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m137.9/137.9 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tenacity>=8.0.2 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (8.5.0)\n",
      "Collecting tomlkit>=0.11.3 (from dlt[lancedb])\n",
      "  Downloading tomlkit-0.13.0-py3-none-any.whl (37 kB)\n",
      "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (4.12.2)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (2024.1)\n",
      "Requirement already satisfied: pyarrow>=12.0.0 in /usr/local/lib/python3.10/dist-packages (from dlt[lancedb]) (14.0.2)\n",
      "Collecting deprecation (from lancedb)\n",
      "  Downloading deprecation-2.1.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting pylance==0.14.1 (from lancedb)\n",
      "  Downloading pylance-0.14.1-cp39-abi3-manylinux_2_28_x86_64.whl (25.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m25.7/25.7 MB\u001b[0m \u001b[31m23.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting ratelimiter~=1.0 (from lancedb)\n",
      "  Downloading ratelimiter-1.2.0.post0-py3-none-any.whl (6.6 kB)\n",
      "Collecting retry>=0.9.2 (from lancedb)\n",
      "  Downloading retry-0.9.2-py2.py3-none-any.whl (8.0 kB)\n",
      "Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (4.66.4)\n",
      "Requirement already satisfied: pydantic>=1.10 in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.8.2)\n",
      "Requirement already satisfied: attrs>=21.3.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (23.2.0)\n",
      "Requirement already satisfied: cachetools in /usr/local/lib/python3.10/dist-packages (from lancedb) (5.3.3)\n",
      "Collecting overrides>=0.7 (from lancedb)\n",
      "  Downloading overrides-7.7.0-py3-none-any.whl (17 kB)\n",
      "Requirement already satisfied: numpy>=1.22 in /usr/local/lib/python3.10/dist-packages (from pylance==0.14.1->lancedb) (1.25.2)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.3->dlt[lancedb]) (0.43.0)\n",
      "Requirement already satisfied: six<2.0,>=1.6.1 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.3->dlt[lancedb]) (1.16.0)\n",
      "Collecting gitdb<5,>=4.0.1 (from gitpython>=3.1.29->dlt[lancedb])\n",
      "  Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting ply (from jsonpath-ng>=1.5.3->dlt[lancedb])\n",
      "  Downloading ply-3.11-py2.py3-none-any.whl (49 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.6/49.6 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: python-dateutil>=2.6 in /usr/local/lib/python3.10/dist-packages (from pendulum>=2.1.2->dlt[lancedb]) (2.8.2)\n",
      "Collecting time-machine>=2.6.0 (from pendulum>=2.1.2->dlt[lancedb])\n",
      "  Downloading time_machine-2.14.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34 kB)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (2.20.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->dlt[lancedb]) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->dlt[lancedb]) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->dlt[lancedb]) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->dlt[lancedb]) (2024.7.4)\n",
      "Requirement already satisfied: types-setuptools>=69.1.0 in /usr/local/lib/python3.10/dist-packages (from requirements-parser>=0.5.0->dlt[lancedb]) (70.3.0.20240710)\n",
      "Requirement already satisfied: decorator>=3.4.2 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb) (4.4.2)\n",
      "Collecting py<2.0.0,>=1.4.26 (from retry>=0.9.2->lancedb)\n",
      "  Downloading py-1.11.0-py2.py3-none-any.whl (98 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.7/98.7 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython>=3.1.29->dlt[lancedb])\n",
      "  Downloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
      "Installing collected packages: ratelimiter, ply, makefun, tomlkit, smmap, simplejson, semver, py, pathvalidate, overrides, orjson, jsonpath-ng, hexbytes, giturlparse, deprecation, time-machine, retry, pylance, gitdb, pendulum, lancedb, gitpython, dlt\n",
      "Successfully installed deprecation-2.1.0 dlt-0.5.1 gitdb-4.0.11 gitpython-3.1.43 giturlparse-0.12.0 hexbytes-1.2.1 jsonpath-ng-1.6.1 lancedb-0.10.1 makefun-1.15.4 orjson-3.10.6 overrides-7.7.0 pathvalidate-3.2.0 pendulum-3.0.0 ply-3.11 py-1.11.0 pylance-0.14.1 ratelimiter-1.2.0.post0 retry-0.9.2 semver-3.0.2 simplejson-3.19.2 smmap-5.0.1 time-machine-2.14.2 tomlkit-0.13.0\n"
     ]
    }
   ],
   "source": [
    "!pip install \"dlt[lancedb]\" lancedb\n",
    "\n",
    "import lancedb\n",
    "from dlt.destinations.impl.lancedb.lancedb_adapter import lancedb_adapter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VzmVW9d5F1aO"
   },
   "source": [
    "**2. Initialize `Notion` verified source**:\n",
    "\n",
    "This command sets up a pipeline that extracts data from the Notion verified source and loads it into a LanceDB destination. You can check what it has loaded in `Files`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XpuB8Nz9GaW2",
    "outputId": "db0000ec-b7d6-4d88-8e34-04e8b0ce2e92"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking up the init scripts in \u001b[1mhttps://github.com/dlt-hub/verified-sources.git\u001b[0m...\n",
      "No files to update, exiting\n"
     ]
    }
   ],
   "source": [
    "!yes | dlt init notion lancedb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UYMWW_Kx3G2g"
   },
   "source": [
    "**3. Import the `dlt.source` that fetches databases from Notion**:\n",
    "\n",
    "Note that we're also defining a `dlt.transformer` function that allows you to manipulate data from a `dlt.resource`. The reason is to pass clean table data to the LanceDB adapter later, without any metadata that notion_databases yields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lT6uxPdM3a1-"
   },
   "outputs": [],
   "source": [
    "from notion import notion_databases\n",
    "\n",
    "# Retrieve all data from specified Notion databases\n",
    "notion_data = notion_databases(\n",
    "    database_ids=[{\"id\": userdata.get(\"NOTION_DATABASE_ID\")}],\n",
    "    api_key=userdata.get(\"NOTION_AUTHENTICATION\"),\n",
    ")\n",
    "# Since `notion_databases` is a dlt.source, we extract the table we have in Notion as a dlt.resource\n",
    "# The `resources` attribute of a dlt.source object contains all the tables in the source.\n",
    "# In this case, we're interested in the 'Reddit-summaries' table.\n",
    "table_resource = notion_data.resources[\"Reddit-summaries\"]\n",
    "\n",
    "\n",
    "# We only need the table data without the metadata, so we use a dlt.transformer which can process yield results from a dlt.resource\n",
    "# The `dlt.transformer` decorator is used to define a function that transforms data from a dlt.resource.\n",
    "# The `data_from` parameter specifies the dlt.resource that the transformer function will process.\n",
    "@dlt.transformer(data_from=table_resource)\n",
    "def get_only_properties(entries):\n",
    "    # This function iterates over the entries in the 'Reddit-summaries' table.\n",
    "    # For each entry, it extracts the 'ID', 'URL', 'Summary', 'Title', and 'Created_utc' properties.\n",
    "    # It then yields a dictionary containing these properties.\n",
    "    for entry in entries:\n",
    "        id = entry[\"properties\"][\"ID\"][\"rich_text\"][0][\"plain_text\"]\n",
    "        url = entry[\"properties\"][\"URL\"][\"url\"]\n",
    "        summary = entry[\"properties\"][\"Summary\"][\"rich_text\"][0][\"plain_text\"]\n",
    "        title = entry[\"properties\"][\"Title\"][\"title\"][0][\"plain_text\"]\n",
    "        created_utc = entry[\"properties\"][\"Created_utc\"][\"rich_text\"][0][\"plain_text\"]\n",
    "        yield {\n",
    "            \"Title\": title,\n",
    "            \"ID\": id,\n",
    "            \"URL\": url,\n",
    "            \"Summary\": summary,\n",
    "            \"Created_utc\": created_utc,\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h8AKhQQhAU7i"
   },
   "source": [
    "**4. Create and run your `dlt` pipeline with `LanceDB` as destination**:\n",
    "\n",
    "`LanceDB` has an integration with `dlt`. All you need to do is just to pass the data with the column you want to embed to the adapter and run the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 298,
     "referenced_widgets": [
      "77a2ca026cb3472ebcb29ba3388f1222",
      "e025aba02ca24b9cad67222213559661",
      "4f9fc618fe5c4f8183a86761d8c2aed8",
      "bb21b7507e1448b68507e211739e31f4",
      "21f436303731479580a679c1132cb969",
      "aee20a941f084eada5cf07526ff3ab78",
      "eced3c811b6f4c1b8e193d30b1208c2e",
      "0926c9946c43466f84876fdbc2b82502",
      "3ee5c6c5f2ab499bb0394542633234ff",
      "b54bcf2674a64afd847ab2faa504b527",
      "b6e06fc29c8a42f3bcbc550970329bbb",
      "e9d27ad3e44b4fcc91da146cf328cadc",
      "14f73de57c5b4cb2b4cce8f9be648f2a",
      "e7e656355c5841e282f144eb64864602",
      "6e0d3c365a744651a8be5f46af36d2e2",
      "369cb5adece640bc825286127442be5c",
      "bff70a9bfad04cda8a80be074175d2ab",
      "dc8265925c1a4fd0a689f5e7d85c625e",
      "65f5f93235f049cdb3dee05f05c958f6",
      "fc56d437fe384c179f4596dda0827b03",
      "fabec3c9424c405b86581778a6b002eb",
      "75ae596e85ab416d81bcd99f84a823d7",
      "9001b488695e4b0ca4ffcf3c2cbc91cd",
      "72a3ce2467df4b6c842a9b5205a5842b",
      "be9afddf7e24446d98e82cae92252bbd",
      "6999fa42e38c4d509fb769b0d2eb4c62",
      "e139dbba011b4805bb353ecfd233782f",
      "58f7e1bc6f744876b60d23dfdc0a1bc9",
      "ba5b48b037af40bdb0c7ac653cb87e08",
      "fac97aa760714e548546c0517f4a344c",
      "a67641b026d245faa1905c433d52ee1d",
      "9e3039190fb24bb2b2cbe35208c57500",
      "417f583a7eb942ee89ddbe1d6b7677d8",
      "91e876b13ce042499dfbe0ad9ec218d0",
      "5058364067b94a02a8bfe05f25f94f48",
      "8fcc060191eb49ea8c897db976e28460",
      "85e09622b29e4ed6b305f43f32e0f6e4",
      "f2cad80c151d4e1ea860721cef74de63",
      "d7094a3b71bc47518f58154930956e6d",
      "58368244f66d44038c21d9ec47af0e14",
      "47de4c8c974c40a3aa3495813eda2d6b",
      "f9203d563a4b478eb614a00102b22f5f",
      "6a907cd9081440dd83b8fa168ba81ac6",
      "06cc09c60ee147bd94f8be5719b92cdf",
      "ad04341154094cd591bc56c261697ffd",
      "63fc11db5a424e91b0994e23e62d3c53",
      "461348a0ce604d3d83c34363f696b39e",
      "8a54a767703649dfb5faa91a836244fe",
      "ecb6e93dc823422cb7b40f6ce5505dc9",
      "0a15c6edfb934ef0859c6e4594cbf559",
      "aa2b006c6dac47f48564e902ed0a579f",
      "1905b89c686942748a778d3f62b941d2",
      "2180a48368db4433b6e3f22b5f013e28",
      "5afb54388af044eb8d7828d66e7d74e6",
      "bce09c307f89497ba18371cac5a1b89a",
      "d8c5a951d0cd4a41a4bede3a2481cec0",
      "ca13ab5d0bb74db8a3d118a2bbd1600b",
      "acf8136c8650476ea4f842bb98c70d14",
      "1c4e96d5102c4a4a8d7b592000eac46d",
      "5123768bb2fd4012a4e632dc82b6174a",
      "39cfdb3f802a4d50ac8ac99c48f1c39f",
      "2ee64da30ab341c59352e9ff24c5b6e0",
      "5719bad4978e45ebaaa28c8e52a7d7b0",
      "e803522efe104844838d4f7fe5dad8ea",
      "727a0a053d8d4f96b94880d4e941e139",
      "a4139ac2aca947c698fc065fd4d6910a"
     ]
    },
    "id": "2yU1PrrFAf_a",
    "outputId": "3041c761-5d6f-4e50-e8b0-757337b03413"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77a2ca026cb3472ebcb29ba3388f1222",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/366 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e9d27ad3e44b4fcc91da146cf328cadc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9001b488695e4b0ca4ffcf3c2cbc91cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/711k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "91e876b13ce042499dfbe0ad9ec218d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/125 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad04341154094cd591bc56c261697ffd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/743 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d8c5a951d0cd4a41a4bede3a2481cec0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/133M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline reddit_lancedb_pipeline load step completed in 5.95 seconds\n",
      "1 load package(s) were loaded to destination LanceDB and into dataset reddit_top_posts\n",
      "The LanceDB destination used <dlt.destinations.impl.lancedb.configuration.LanceDBCredentials object at 0x78e4b4584670> location to store data\n",
      "Load package 1721294703.4760716 is LOADED and contains no failed jobs\n"
     ]
    }
   ],
   "source": [
    "# Note that we're using open-source tools, so we don't provide any keys here\n",
    "credentials = {\n",
    "    \"uri\": \"reddit_summaries.lancedb\",\n",
    "    \"api_key\": \"\",\n",
    "    \"embedding_model_provider_api_key\": \"\",\n",
    "}\n",
    "\n",
    "# Create your dlt pipeline with LanceDB as destination\n",
    "lancedb_pipeline = dlt.pipeline(\n",
    "    pipeline_name=\"reddit_lancedb_pipeline\",\n",
    "    destination=dlt.destinations.lancedb(\n",
    "        credentials=credentials,\n",
    "        embedding_model_provider=\"huggingface\",\n",
    "        embedding_model=\"BAAI/bge-small-en-v1.5\",\n",
    "    ),\n",
    "    dataset_name=\"reddit_top_posts\",\n",
    ")\n",
    "\n",
    "load_info = lancedb_pipeline.run(\n",
    "    lancedb_adapter(get_only_properties, embed=[\"Summary\"]), write_disposition=\"replace\"\n",
    ")\n",
    "print(load_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y_o2oTBKtBT_"
   },
   "source": [
    "**3. Query your data:**\n",
    "\n",
    "This script connects to a LanceDB database, retrieves data from a specific table, searches for a query within the table, and converts the search results to a pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GxjKxGWvqLOL",
    "outputId": "ab869365-b9dd-49ca-eb4b-3cf980720241"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I was 21 when my fiance asked me to marry him. We were only engaged for 6 months before the inncident. My middle oldest sister, lets call her Nicky, was a very cold person. She only ever opened up to my fiance as she said she saw him as a brother. She and I never saw eye to eye, I loved her dearly because she was my sister but didn't like her as a person. The night was going smoothly until Nicky spotted a guy across the room whom she claimed she wanted to \"climb like a tree\" She walked over to him and within a few minutes she was back and she had a sour expression on her face. She then told me the guy didn't want her number but he wanted mine instead. I don't remember what happened next as I blacked out and the next morning I woke up on a hard sofa, my head pounding. When I came to, I realised I was in Nicky's friends house and my phone was sitting on the glass table in front of me, but it was flat. I tried to explain that my phone went flat but he then went on screaming about how could I cheat on him. Nicky told her ex fiance that she had slept with him multiple times. When he found out he left and never returned any of her calls or texts. Nicky's mother kicked her out and threw her things out. She was homeless and single in less than a day and a half. She tried everything to get her fiance back and her family back. But they all chose Nicky and her side and left her to fend for herself. She has not spoken to any of them in two years and doesn't know if she can ever forgive them. If you would like to talk to her about her story, please contact her on 020 3615 9090. For confidential support call the Samaritans in the UK on 08457 90 90 90, visit a local Samaritans branch or click here for details. In the U.S. call the National Suicide Prevention Line on 1-800-273-8255. \n",
      "\n",
      "\"I never wanted kids, but was never adamantly against having one\" \"My wife's sister died. All of a sudden \"family\" is SUPER important to my wife\" \"I've read every book. I've worked shifts 6 days a week for a decade to pay for tens of thousands (probably 100,000's) of therapy, behaviorists, counseling, classes\" \"Guess what? Grandma and grandpa say the kid is \"too much\". They haven't helped for more than a day a month in almost 7 years. And here I am - on reddit on my laptop, tethered to my phone in a park after dark\" \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Connect to the database and retrieve the data\n",
    "db = lancedb.connect(\"reddit_summaries.lancedb\")\n",
    "table = db.open_table(\"reddit_top_posts___get_only_properties\")\n",
    "\n",
    "query = \"marriage\"\n",
    "result = table.search(query).limit(2)\n",
    "df = result.to_pandas()\n",
    "\n",
    "# Now you can print or manipulate the DataFrame\n",
    "texts = df[\"summary\"]\n",
    "\n",
    "for text in texts:\n",
    "    print(text, \"\\n\")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
